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developments is crucial and significantly influences the assessment results. The methodology, which 
consists of two processes: a pre-selection of suitable C&I by the researchers and a final selection by 
regional bioenergy experts in a multi-stakeholder forum, was applied in a Scottish region (Tayside & Fife). 

The paper concludes that the methodology provides a transferable approach to systematically select 
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1. Introduction 


This paper presents a methodology for participatory selection of 
criteria and indicators (C&I) for bioenergy developments. The meth- 
odology was applied in a case study area in Scotland (Tayside and Fife). 
Sheppard and Meitner [1] and others [2-4] highlight that C&I reflect 
issues of concern of stakeholders and experts of an industry and can 
be used to measure and communicate the sustainability of projects or 
progress on sustainable development. In the ‘Guidelines for Applying 
Multi-Criteria Analysis to the Assessment of Criteria and Indicators' [5], 
C&I are stratified into four levels. The first level comprises of a 
principle, which is described as a “fundamental truth or law as the 
basis of reasoning or action”. Principles justify the chosen criteria, 
indicators and verifiers. On the second level, a criterion is defined as a 
“principle or standard that a thing is judged by”. Criteria enhance the 
meaning and operability of principles, however do not measure 
performance. On the third level, an indicator is described as a variable 
or component “used to infer the status of a particular criterion”. 
Indicators are comprised of specific information about a criterion, and 
can be measured by verifiers, which are defined as “data or informa- 
tion that enhances the specificity or the ease of assessment of an 
indicator” [5] (p.11 et seq.). Waage et al. [6] and Madlener et al. [7] use 
a similar hierarchy by describing that criteria are operationalized with 
indicators. For example, the criterion ‘climate change’ can be oper- 
ationalized by the indicator ‘CO2 equivalent emissions’. However, in 
literature about decision making it can be observed that the term 
‘indicator’ is sometimes synonymously used with ‘criterion’ and vice 
versa [8,9]. 

Vera [3] stress that if taken together C&I can provide an 
overview of the whole energy system, including interlinkages 
and trade-offs among various aspects of sustainable development, 
as well as long-term implications of current decisions and beha- 
viors. Progress or lack of it towards sustainable development can 
be measured by observing changes in C&I values over time [10], 
which allows the carrying out of focused measures for imp- 
rovement to avoid adverse effects in specific areas, if required. 
Furthermore, they can be appropriate tools for communicating and 
promoting dialogue with respect to energy issues related to 
sustainable development to stakeholders, policymakers and the 
public [3]. Additionally, Olsthoorn et al. [11] point out that the high 
complexity level of an issue, for instance the natural environment, 
demands the need of appropriate C&I. The selection of appropriate 
C&I is crucial to assess a project or development, because this 
choice significantly influences the results of an assessment [12]. 
Therefore, it was emphasized that in the sustainability context, C&I 
selection becomes a “delicate process of translation from socio- 
environmental agreements to specific observed properties of a 
complex system” [13] (p. 2562), and that the chosen C&I are the 
result of the social and political framework at a particular period in 
history [14]. 

In this paper a methodology for participatory selection of C&I for 
bioenergy developments is described. The methodology was also 
applied in the mentioned case study area (Tayside & Fife). First, 
literature on methodologies to select C&I for decision-making in the 
sustainability assessment field in general, as well as the energy and 
bioenergy fields in particular, was reviewed. Then, an appropriate 
methodology for participatory C&I selection was carried out starting 
with identifying C&I from literature against which sustainability in the 
bioenergy and the wider energy sector have previously been assessed. 
This was followed by the identification and discussion of requirements 
to select appropriate C&I from literature. Based on these two generic 
steps, the methodology was specifically applied in the case study area, 
which involved multiple steps. First, scenarios and alternatives were 
developed. Second, most suitable requirements for C&I selection for 
the case study were determined. Then, the sustainability C&I identified 
from literature were pre-selected in accordance with the determined 


requirements. The next step was to present the resulting short-list 
of C&I to identified bioenergy experts for discussion in a multi- 
stakeholder forum. In this forum the participants were asked to reach 
a consensus on the most suitable C&I for the case study by taking 
account of the same requirements for selection as in the pre-selection 
process. In this paper, the results of the application of the participatory 
C&I selection methodology are also discussed before conclusions 
are drawn. 


2. Methodology for participatory C&I selection 


In this section methodologies to select C&I for decision-making in 
the sustainability field and the energy and bioenergy sectors from 
relevant literature are briefly described. In general, Starkl and 
Brunner [15] recommend that integrative assessments of sustain- 
ability should not be based on prescribed or standardized C&I and 
rather be specifically selected on a case to case basis. Due to the sheer 
volume of C&I, which can be found in literature, several authors of 
academic papers propose to identify and pre-select C&I based on 
relevant literature, followed by a process of verification and/or 
refinement by stakeholders [1,16,17]. Gilmour et al. [18] followed 
this approach and additionally developed new case-specific C&I. 
First, the C&I pre-selection process took place. Then, key stakeholders 
were individually interviewed to verify and refine pre-selected C&l. 
In this process stakeholders were asked to take account of the 
specific drivers, aspirations and objectives of the development. 
In addition to the interviews a document analysis to refine and 
verify potential indicators, as well as to identify associated data 
availability was undertaken. 

Sheppard and Meitner [1], Gilmour et al. [18] and Graymore 
et al. [16] suggest using C&I or selection frameworks, which are 
specific to a country or region and/or which are well-established in 
the industrial sector. For instance, in Graymore et al. [16], a project 
undertaken in South-West Victoria (Australia) is described, for 
which a set of C&I applicable for tracking progress towards 
sustainable development was identified. First, they identified a vast 
array of C&I already in use for measuring sustainability at a global, 
regional and local scale through an extensive literature review. 
Using this ‘global’ set of C&I a ‘filtering’ process’ was carried out to 
refine the C&I set by involving stakeholders representing regional 
organizations and communities from their case study area. 

Following the methodology recommendations from the reviewed 
literature, the methodology presented in this paper consisted of both, 
generic and case-study specific parts. First, C&I from literature were 
identified against which sustainability in the bioenergy and the wider 
energy sectors have previously been assessed (Section 2.1). Then, 
requirements to select appropriate C&I from literature were identified 
(Section 2.2). Based on these two generic steps, the methodology was 
specifically applied in the case study area involving the main steps: 
development of scenarios and alternatives (Section 3.1), selection of 
requirements for C&I selection (Section 3.2), C&I pre-selection by the 
researchers (Section 3.3) and final C&I selection by regional bioenergy 
experts (Section 3.4). In the subsequent sections these processes are 
described and discussed in detail. 


2.1. Identification of sustainability C&I in literature 


Employing C&I for decision making has been extensively addressed 
in academic papers and other publications covering the sustainability 
assessment field in general, as well as the energy and bioenergy fields 
in particular. In Table 1C&lI from reviewed literature are summarized. 
In general, it can be observed that C&I in literature are either grouped 
into the broad categories of social, economic and environmental 
[2,19-22] or an additional forth aspect (technical) is common to 
evaluate energy systems or scenarios in particular [7,9,12,17,23,24]. 
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Table 1 
Summary of C&I identified in literature. 


C&I related to: 


Environmental C&I: 
NO, emissions 


Air quality or other non-GHG emissions (e.g. CO emissions, SO2 emissions, particles emissions) 


CO» emissions 


GHG emissions or climate change (e.g. total carbon savings) 
Land take, use or requirement, land use change or effective land use 


Direct or indirect land conversion caused by biofuel plantation (e.g. visual impacts, quality of landscape, effects on biodiversity and 


Source: 


[20,24,27,30-33] 
[3,7,8,12,17,19,24,31] 
[8,12,17,20,24,26,27,30,32- 
38] 

[3,9,19,25,31,39] 
[12,24,38,40] 
[3,7,9,17,19,39] 


ecosystems, species protection, deforestation, soil quality and protection, crop management and agrochemical use) 


Ecological justice 

Adaptability 

Noise 

Water consumption 

Water management or quality 

Waste generation or management 

Rational use of resources or natural resource efficiency 
Economic C&I: 

Investment costs 

Operational or maintenance costs 


Costs and economic viability (e.g. net present value (NPV), payback periods, life cycle cost, microeconomic sustainability) 


Energy prices/costs for end-users 


Macroeconomic sustainability, economic development, stability, benefit or output, balance of trade or payments 


Technological advantage or diversification of technologies 


Resources or fuel availability or import dependency (e.g. import dependency, resource availability, regional self-determinacy, energy 


security, reliability of electricity supply) 
Technical C&I: 
Efficiency 


[7,17] 

[7,17] 
[7,17,19,24,41] 
[9,12,38] 
[3,7,17,19,39] 
[3,18,19,39,41,42] 
[7,8,17,19] 


[8,12,18,20,24,25] 
[7,8,17,20,24,38] 
[19,24,25,37,40] 
[3,9,20,24] 
[17,19,39,40] 

[3,7,17] 
[3,7,9,12,17,38,40,43] 


[3,8,9,12,20,24-29,38] 


Health or safety of energy systems [3,24,39] 
Maturity or knowledge level of the energy generation technology in use [24,25,34,35] 
Reliability of energy systems as well as their technological capability and limitations [24,28,30,44] 
Electricity generation potential or average annual availability [12,31] 
Primary energy ratio [24,34,35] 
Energy intensities of the industrial, household and commercial level (e.g. industrial, agricultural, service/commercial, household and [3,20] 
transport energy intensities) 
Energy balance [19,42] 
Social C&I: 
Job or employment creation or opportunities [7,8,12,17- 
20,24,25,27,28,30,31,35] 
Job income or working conditions [12,19,20,39] 
Access or rights to use land, water and other natural resources [19,39] 
Food security in the context of bioenergy generation and biofuel supply [19,39] 
Decision-making, participation, responsibility, confidence or empowerment as well as planning, management or monitoring [17-19] 
Wider social C&I (e.g. respect for human rights and minorities, standard of living) [19,39] 
Accessibility, affordability or disparities [3,20] 
Social or cultural cohesion, acceptability or benefits (e.g. social justice, effect on public spending, social impacts) [7,9,17-19,24] 
Compliance with laws [19] 


The identified C&I (Table 1), which built the basis for the case study’s 
C&I pre-selection (3.3), are also divided in these four categories 
following the alternative ordering of “environmental”, “economic”, 
“technical”, and “social” aspects. 

In sustainable energy decision making it can be observed that 
‘CO emission’ and ‘Investment cost’ are the most common 
environmental and economic aspects, respectively [24]. Further- 
more, ‘Efficiency’ is the most common technical criterion to 
evaluate energy systems according to Jovanovic et al. [20] and 
others [8,24-29] and in their evaluation of the impacts of power 
plants, Chatzimouratidis and Pilavachi [30] state that in the energy 
sector, ‘Job creation’ is considered as the key social aspect used for 
sustainability assessments. 

It can be concluded that the vast number of C&I makes it 
imperative to take a systematic approach to select C&I for individual 
projects or case studies. A systematic approach using requirements 
for C&I selection is described in the subsequent sections of this paper. 


2.2. Requirements for criteria and indicators selection from literature 


Requirements for C&I selection enable a ‘filtering process’ as 
mentioned previously. These requirements can be found in literature 


about the sustainability assessment field in general, as well as about 
the energy and bioenergy fields particularly. These types of literature 
were considered in order to select the most appropriate set of 
requirements. 


2.2.1, Requirements for C&I selection: sustainability assessment field 

Requirements for sustainability C&I selection have been widely 
described in literature about sustainability assessment in general. 
Graymore et al. [16] used three ‘filters’ ensuring: (a) C&I relevance to 
the region, (b) consideration of both trend and condition (time 
dynamic C&I) and (c) consideration of the relationships between C&I 
to avoid an isolated perspective on C&I. First, they asked stakeholder 
organizations and community representatives to select most suitable 
C&I from a set of pre-identified C&I considering regional relevance 
(first filter). Then, the researchers filtered the remaining C&I with 
respect to practicality and reliability and GIS software applicability 
(second filter). Finally, based on this reduced set of C&I the third filter 
took relationships and indicator independencies into account to 
develop a small number of highly influential C&I. 

Singh et al. [21] provide the following dimensions of C&I 
measurements: (a) sustainability aspect, (b) technique/method 
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employed (quantitative/qualitative, subjective/objective, cardinal 
/ordinal, one-dimensional/multidimensional), (c) type of compar- 
ison (across space (cross-section), time (time-series) or in an 
absolute/relative manner), (d) input (‘means’) or output (‘ends’) 
focus, (e) clarity and simplicity, (f) data availability across time and 
space, (g) flexibility (e.g. for change). These dimensions rather help 
to classify and evaluate C&I measurements than to select appro- 
priate C&I. However, since they address C&I measurements, they 
can be summarized into the requirements practicality, simplicity 
and reliability, whereas the latter covers the reproducibility of 
assessment results. 

According to Baker et al. [45] C&I should be: (a) able to dis- 
criminate among alternatives, (b) complete in regard to all goals, 
(c) operational and meaningful to understand the impacts of the 
alternatives, (d) non-redundant and (e) few and manageable in 
number. In other words C&I should measure important issues, 
comprehensively and in a meaningful way, hence they should be 
relevant keeping the problem’s goal in mind. Comparability must 
be ensured, as well as a high degree of independency of C&I. 
The notion that C&I should be few in number to increase under- 
standability is rather an overall requirement than an individual 
selection requirement to decide, whether or not a criterion and its 
indicator(s) should be chosen. 

For another project [46] sustainability C&I were selected and at 
a later stage reviewed by stakeholders based on the following 
specifications: (a) comprehensiveness, (b) applicability to each 
alternative, (c) transparency, (d) tractability of data, and (e) 
practicability with respect to available time and resources. 
The first two specifications can be summarized into the require- 
ment relevance and the last two into practicality. Transparency is 
described as the stakeholders’ ability to understand C&I or in other 
terms it describes the simplicity of C&I. This simplicity of C&I is 
also emphasized by Olsthoorn et al. [11] (p.457) who suggest as a 
general guideline that C&I “should be as simple as possible and 
only as complex as necessary”. 

Gilmour et al. [18] interviewed key stakeholders individually to 
verify and refine pre-selected C&I with reference to the specific 
drivers, aspirations and objectives of a specific sustainable water- 
front development. In this process each criterion and its indicator 
(s) were addressed to verify relevance. 

Another participatory approach was taken by Fraser et al. [10] 
who held community focus group meetings to rank C&I according 
to their perceived accuracy and ease of use. These two require- 
ments refer to practicality and simplicity of C&I, respectively. C&I 
can also be selected by simply considering the availability of data, 
i.e. the practicality of C&I can be regarded as the single require- 
ment [13]. 

Lattimore et al. [2] in their paper about sustainable wood fuel 
production outline that C&I should be as broadly applicable as 
possible and be imbedded within a framework of local adaptability. 
Furthermore, they identified C&I based on comprehensiveness and 
effectiveness. These aspects can be covered by the requirements 
practicality and relevance. 


2.2.2. Requirements for C&I selection: energy and bioenergy sectors 

Apart from literature about sustainability assessments in gen- 
eral, requirements for C&I selection have also been described in 
literature specific to the fields of energy and bioenergy. In a survey 
about sustainability C&I for bioenergy systems [19], experts were 
asked to rate C&I according to relevance, practicality, reliability and 
importance. Since determining the importance of C&I is typically 
covered by separate decision making processes (e.g. [47-49], this 
requirement was excluded for further considerations in the C&l 
selection process described in this paper. C&I relevance is the first 


requirement described and refers to relevance in regard to the 
concept of sustainable bioenergy systems, as well as the contribu- 
tion of this aspect’s assessment to an improved understanding of 
the sustainability context. In terms of practicality, experts were 
asked in the case study described by Buchholz et al. [19] to consider, 
whether existing scales and/or measurement units and measure- 
able threshold values exist for the individual C&I. In addition, the 
ease of obtaining data and measuring C&I in a cost, time and/or 
resource effective manner are mentioned attributes for this require- 
ment. How reliable C&I assessment results are and whether these 
results can be reproduced, are considerations in regard to the 
reliability requirement. Further considerations for this specific 
requirement are: high uncertainty attached to C&I and the ease of 
reaching consensus. 

Wang et al. [24] state that C&I development and selection 
requires parameters associated with reliability, appropriateness, 
practicality and limitations of measurement. In this context, 
appropriateness refers to C&I relevance and limitations of mea- 
surement could be part of the practicality requirement for C&I. 
In general the authors share the opinion that a low quantity of 
C&I is beneficial to the evaluation of energy systems process. 
Furthermore, they establish principles for decision-makers to 
select C&I. The following principles are suggested: (a) consistency, 
(b) independency, (c) measurability, (d) comparability. The first 
principle suggests consistency with the decision-making objective 
and therefore relevance. The independency principle demands 
a criterion and its indicator(s) to be excluded, if an inclusion 
relationship at the same level exists. Furthermore, a criterion and 
its indicator(s) should allow reflection of the alternatives’ perfor- 
mances from a different perspective. The third principle requires 
C&I to be quantitatively or qualitatively measurable and hence to 
be practical. Finally, the comparability principle should ensure 
distinction between alternatives, whereas it also suggests normal- 
ization processes to compare between C&I results. 

In a paper about sustainability indicators in the electricity sector 
[12] 15 selection requirements are grouped under three major 
topics: (a) conception, (b) application and (c) consistency. The first 
requirement in the group ‘conception’ is adhesion, which is about a 
direct relationship of C&I to the analyzed aspect. This requirement 
could fall under the relevance requirement. Feasibility and validity, 
the next two requirements in this group comprise of aspects 
covering practicality or reliability issues of C&I. With respect to 
practicality, data availability and the feasibility of additional data 
collection is also outlined in a paper, which provides energy 
indicators for sustainable development [3]. The requirement clarity 
in Rovere et al. [12] is subject to the group’s last requirement 
simplicity. In the next group ‘application’ the requirements: sensi- 
tivity and temporality and ease cover the practicality aspect of C&l, 
because they represent their ability for trend and temporal analyses 
and the difficulty level of their practical application. The spatiality 
requirement covers comprehensiveness of C&I and hence their 
relevance. The remaining requirement in this group is reliability 
and is described as the ability to capture both, positive and negative 
issues in an unbiased manner. In the final group ‘consistency’ the 
requirement discernment reflects an aspect of comparability, 
whereas equilibrium refers to the independency of C&I. Addition- 
ally, the requirement verifiability comprises of the capacity of C&I to 
reproduce data and therefore addresses an aspect of reliability. 
Relevance is separately represented in this group as the remaining 
requirement. This requirement is also addressed in a paper about 
technological energy priorities and the multi-criteria decision- 
making approach [25], in which the problem’s formulation, a 
country’s specific energy characteristics, as well as its development 
needs and perspectives are considered as determinants influencing 
C&I and their performances. These determinants could be summar- 
ized into the relevance requirement. 
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3. Application of methodology in case study area 
3.1. Scenario description 


Two scenarios, ‘A1-2’ and ‘A3-4’, each with a centralized and a 
decentralized bioenergy strategy for the Scottish Region Tayside 
& Fife were developed (Table 2). However, the scale of energy 
generation differed between the scenarios, as well as the alter- 
natives. For both of the first scenario’s alternatives the total 
electrical outputs were assumed to be 100 MWe. Also, a second 
scenario with 10 MWe electrical outputs for each of this scenario’s 
alternatives was developed. For this second scenario the diversify- 
ing parameters and hence the complexity level of comparison was 
significantly reduced by assuming the same source and type of fuel 
to be used for both alternatives. The reason for using these specific 
scenarios and alternatives for the research case study was to allow 
the decision making process to be independent not just from the 
scale of the bioenergy developments, but also from the complexity 
level of the comparison. In other words by establishing and 
presenting to bioenergy experts scenarios with varying levels of 
differences in specifications — one in which all specifications differ 
and one in which only a single specification distinguishes the two 
alternatives - the decision making processes were less influenced 
by the complexity of the comparison. Furthermore, the presented 
range of scenarios and alternatives was intended to demonstrate 
the broad application potential of this participatory approach to 
select sustainability C&I in the bioenergy field. 


3.2. Requirements for C&I selection for case study 


Based on the literature review described previously (Section 
2.2), the final set of requirements for C&I selection for the case 
study was chosen as shown in Table 3. Each requirement was 


Table 2 
Case study scenarios and alternatives. 


described for clarification based on the literature review. In line 
with this set of requirements, C&I were pre-selected first and 
then presented to regional bioenergy experts to decide which 
sustainability C&I should be employed for the case study. 


3.3. Pre-selection of C&I for case study 


Based on the identified sustainability C&I (Section 2.1) and 
the selection requirements (Section 2.2), a number of sustain- 
ability C&I were pre-selected prior to presenting the resulting 
C&I short-list to regional bioenergy stakeholders to reach 
consensus on them. For this step the case study scenarios as 
described previously had to be kept in mind at each stage of 
the process. The process consisted of a series of sorting steps 
before the most appropriate C&I were chosen. 

First, C&I from literature were sorted by the four categories: 
environmental, economic, technical and social. As a second sort- 
ing step, the C&I were summarized, resulting in the following 
classification: 


- Environmental: GHG Emissions, Air Quality, Water, Waste and 
Other Environmental C&l. 

- Economic: Economic Viability, Regional Energy Self-Sufficiency 
and Other Economic C&I. 

- Technical: Efficiency, Technology, Other Technical C&I. 

- Social: Regional Job Creation, Energy for Households, Employ- 
ment Conditions, Regional Food Security and Other Social C&I. 


At this point it became apparent that some C&I were redundant 
and others could be classified in more than one category. For 
example, due its nature and potential effects on both aspects, C&I 
related to fuel and resource availability and import dependency can 
either be in the economic or in the social category. C&I definitions, if 


Alternative Total electrical output Strategy Type of fuel Source of fuel 
(MWe) 

Scenario 1 

A1: Large-scale bioenergy plant (predominantly for 100 Centralization Single Woody biomass Predominantly from 
electricity generation) strategy location abroad 

A2: 20 Medium-scale bioenergy combined heat and 100 De-centralization Multiple Any biofuel (and related Predominantly 
power (CHP) plants strategy locations technologies) regional sources 

Scenario 2 

A3: Medium-scale bioenergy combined heat and 10 Centralization Single Any biofuel (and related Predominantly 
power (CHP) plant strategy location technologies) regional sources 

A4: 10 Small-scale bioenergy combined heat and 10 De-centralization Multiple Any biofuel (and related Predominantly 
power (CHP) plants strategy locations technologies) regional sources 


Table 3 
Requirements for C&I selection for the case study. 


Requirement for Description: Source: 

selection: 

Relevance: How relevant are the criterion and its indicator(s) for decision-making for sustainable bioenergy developments? Does [2,12,16,18,19,24,25,45,46] 
the assessment contribute to a better understanding of the sustainability of bioenergy developments? 

Practicality: Do scales and/or measurement units exist? Can data easily be obtained and measured in a cost, time and/or resource [2,3,10-13,16,19,21,24,46] 
effective manner? 

Reliability: How reliable/reproducible are the assessment results? Is there a high uncertainty attached to the assessment results? [12,16,19,21,24] 

Independency: Are the criterion and its indicator(s) independent enough, i.e. do they reflect the performance of alternatives from a 12,16,24,45] 
different viewpoint or do they duplicate other C&I? 

Comparability: Are the criterion and its indicator(s) able to discriminate among alternatives and do they support the performance 12,24,45] 
comparison of alternatives? 

Simplicity: Are the criterion and its indicator(s) easy to understand by all stakeholders? 10,21,46] 
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available from the reviewed papers, were of assistance to eliminate 
redundant C&I or to aggregate similar C&I such as ‘Soil Protection’ 
and ‘Soil Quality’ or ‘Job Creation’, ‘Employment Generation’ and 
‘Contribution to Employment Opportunities’ Creation’. This process 
was characterized by selecting the most case-study relevant criter- 
ion and indicator to represent similar aspects (e.g. ‘Air quality’ and 
the indicator ‘Non-GHG Emissions’). In this procedure the six 
requirements for C&I selection as defined in Table 3 were also 
considered. For example the indicator ‘Net Present Value’ was 
selected to represent ‘Economic Viability’, because cost indicators 
such as ‘Investment Cost’ were basically included in the calculation 
of that indicator. At this stage, C&I were also modified or new C&I 
were developed to take account of topical issues, which were felt to 
be not or not fully addressed in the reviewed literature such as for 
example the indicator ‘Threat for Food Security’, which emphasizes 
and combines issues in regard to food security and land conversion 
caused by biofuel plantation. These topical issues were typically 
characterized by a high relevance in regard to decision-making for 
sustainable bioenergy developments at the time when the case 
study was undertaken. Also, in order to add more case-study- 
specific relevance, indicator units were modified (e.g. ‘Created Jobs/ 
kWh (plant and supply chain specific) for the criterion ‘Job 
Creation’). Based on these changes the final C&I descriptions were 
developed. By again comparing each of the C&I against the six 
requirements for selection as defined previously (Table 3), the final 
pre-selection of C&I was undertaken. In this process, a criterion and 
its indicator were assessed using a three-point scale with ‘low’, 
‘medium’ and ‘high’ scores. The threshold score was the score ‘low’, 
i.e. if a score against a requirement was ‘low’, the assessed criterion 
and its indicator was excluded. Table 4 below illustrates the scoring 
results of the C&I pre-selection process: 

The following two examples illustrate how C&I were scored at 
this pre-selection stage: 


3.3.1. Air quality 

The first scoring against the six selection requirements com- 
prised the criterion ‘Air Quality’ and its indicator ‘Non-GHG 
Emissions’. This criterion and its indicator were considered to be 
highly relevant and to have a high comparison level for the case 
study alternatives. The high score against ‘comparability’ was due 
to the quantitative nature of the criterion’s indicator, which 
enables alternatives to be compared relatively easily. The high 
score for relevance was justified by the aggregation potential of 
the criteria, as it allows the inclusion of several emissions 
(NOx, CO, SOz and particulates). In regard to ‘practicality’, 


Table 4 
Scoring results of the C&I pre-selection for the case study. 


Category Criterion Indicator 

Environmental Air quality Non-GHG emissions 
GHG emissions CO, emission 
Water Water management 
Waste Waste re-processed 
Others Others 

Economic Economic viability Net present value 
Regional energy self-sufficiency Fuel import dependency 
Others Others 

Technical Efficiency Energy efficiency 
Technology Knowledge of the technology 
Others Others 

Social Regional job creation Job creation 


Energy for households 
Employment conditions 
Regional food security 
Others 


Average level of job income 
Threat to food security 
Others 


Number of households supplied 


‘reliability’, ‘independency’ and ‘simplicity’ ‘medium’ scores were 
assigned. Although, often measured for projects of this type, the 
indicator ‘Non-GHG emissions’ and its mentioned aggregation 
potential was regarded as causing potential limitations in regard 
to the first two requirements, because data availability and 
reliability of all the case study’s emissions was assumed to be 
incomplete or limited. Aggregation was also the reason for the 
medium score for ‘simplicity’, because multiple data for one 
indicator was regarded as influencing negatively the understand- 
ability of the criterion and its indicators for all bioenergy experts. 
The criterion and its indicator are influenced by several other C&I 
and thus a ‘medium’ score was assigned to the requirement 
‘independency’. 


3.3.2. Regional food security 

The criterion ‘Regional Food Security’ with its indicator ‘Threat 
to Food Security’ scored ‘high’ against ‘relevance’, because this 
aspect was related to both, the topical food versus-fuel-contro- 
versy, which covers the issue of diverting farmland or crops for 
biofuels production in detriment of the food supply, as well as the 
already existing and predicted increase in competition for biofuels 
resources in the case study area. It was assumed that it would be 
difficult to obtain and measure data in a cost, time and/or resource 
effective manner and predictive data would rather be uncertain, 
due to the early stage of energy crop plantation in Scotland. Hence, 
for both requirements ‘practicality’ and ‘reliability’ ‘medium’ score 
were assigned. The criterion is influenced by other C&I and scored 
‘medium’ against ‘independency’. The criterion allows discrimina- 
tion among the scenarios’ alternatives and therefore scored ‘high’ 
against ‘comparability’. For the last requirement ‘simplicity’ it was 
assumed that not all experts would easily understand the criterion 
and its indicator, which was reflected in a ‘medium’ score. 

Table 5 below illustrates the pre-selected C&I. This table was 
presented to bioenergy experts for the final selection process: 


3.4, Final selection of criteria and indicators by experts in case 
study area 


Similar to other studies in the sustainability and bioenergy 
sector [19,50], participants of the multi-stakeholder forum were 
bioenergy experts rather than members of the general public and 
like in Sheppard and Meitner [1] were selected based on their 
knowledge and expertise in the subject, their history with plan- 
ning processes and based on how much they would be affected by 


Relevance Practicality Reliability Independency Comparability Simplicity 


High Medium Medium Medium High Medium 
High Medium Medium Medium High High 
Low Medium High Medium High High 
High Medium Medium Medium High High 
N/A N/A N/A N/A N/A N/A 
High Medium Medium Medium High Medium 
High Medium Medium Medium High Medium 
N/A N/A N/A N/A N/A N/A 
High High High Medium High High 
High Medium Medium High High Medium 
N/A N/A N/A N/A N/A N/A 
High Medium Medium Medium High High 
Low Low Low Medium Medium High 
Low Medium High Medium Medium High 
High Medium Medium Medium High Medium 


N/A N/A N/A N/A N/A N/A 
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Table 5 
Pre-selected C&I for the case study. 
Indicator Unit 


Category Criterion 


Environmental GHG emissions CO» emissions 


Air quality Non GHG- kg/kW h (operational plant specific) 
emissions 
Waste Waste re- % of Scottish waste re-processed to 
processed fuel (per annum) 
Economic Economic viability Net present £/kW h (operational plant specific) 
value 
Regional energy Fuel import % of imported feedstock (operational 
self-sufficiency dependency plant specific) (per annum) 
Technical Efficiency Energy % (operational plant specific) 
efficiency 
Technology Knowledge of % (worldwide) 
technology 
Social Regional job Job creation Created jobs/kW h (plant and supply 
creation chain specific) 
Regional food Threat to food 
security security crop plantation 


the scenarios’ alternatives. It was ensured that all interests and 
aspects of sustainability in the field were covered by equal 
proportions of participating representatives from different back- 
grounds, which as outlined by Voinov and Bousquet [51] was also 
intended to increase public acceptance and credibility of the 
forum’s outcomes. A relatively balanced composition of partici- 
pants from local authorities, the regulative body, the business 
support agency, environmental protection, harvesting and supply, 
sawmilling, bioenergy production, agriculture, forestry and waste 
management was achieved by nonproportional quota sampling 
[52]. In total 13 regional bioenergy experts participated in the 
multi-stakeholder forum. 

The invited participants were asked to reach consensus about 
the final set of sustainability C&I for the case study scenarios and 
alternatives. It was explained that the selection of the most 
appropriate C&I is crucial to assess sustainability, because any 
aspects not included would simply be ignored from the decision 
making process, which again would influence assessment results. 

First, a PowerPoint™ slide explaining the four distinct aspects 
within sustainability that interrelate - environmental, economic, 
social and technical - was presented to the participants of the 
multi-stakeholder forum. After highlighting the goal of the multi- 
stakeholder forum (consensus about C&I selection) and explaining 
the procedure, the requirements for C&I selection along with their 
descriptions (see Table 3) were presented on another Power- 
Point™ slide, of which also hard-copies were handed out. 
The experts were explained that those six requirements were 
used for pre-selection and asked to make their own selection 
choices also based on them. Furthermore, presented slides with 
problem hierarchies for both scenarios allowed participants to 
keep an overview of the relationships between the goal, cate- 
gories, criteria and alternatives (see Fig. 1 as an example). 

The pre-selected C&I (Table 5) were presented to the partici- 
pants grouped by sustainability aspect. First, questions were 
answered and issues were clarified in regard to C&I descriptions. 
Then, the process of final C&I selection itself was carried out by 
directly asking the forum participants to reach consensus about 
the suitability of each pre-selected criterion and its indicator to 
assess the case study’s scenarios and alternatives. The participants 
were encouraged to confirm or make changes to the presented set 
or propose new C&I, again by taking account of the defined six 
requirements for selection. In cases when consensus about pre- 
selected or newly proposed C&I could not be reached, the C&I were 
regarded suitable, when there was evidence that a significant 
majority of forum participants agreed on them. To keep track of 
the communication notes were taken during and after the event. 


kg/kW h (supply chain specific) 


Description 


Emissions of CO2 across bioenergy supply chain 

Emissions of NO,, CO, SO, and particulates from operational bioenergy 
plant(s) 

Waste re-processed to fuel for bioenergy production 


Expected incoming and outgoing cash flows derived to present economic 
values to enable a comparison independent of time 
Percentage of imported feedstock for the bioenergy plant(s) 


Expresses the technology’s ability to convert the primary energy source 
to energy 

Represents the technology's maturity rate, as well as its uptake/adoption 
in the international market 

Number of direct and indirect jobs created at the bioenergy plant(s) and 
along the Scottish bioenergy supply chain 


% of total arable land used for energy Reflects the availability of remaining arable land for food production in 


Scotland 


4. Results and discussion 


The processes of the pre- and final selection of C&I are discussed 
in the following. 


4.1. C&I pre-selection process 


When reflecting on the pre-selection process it can be observed 
that all pre-selected C&I scored ‘high’ against the requirements 
‘relevance’ and ‘comparability’. The reason for that was that 
throughout the process the case study scenarios and alternatives 
were kept in mind. In other words, the steps including sorting and 
classification, modifying of indicator units and the final descrip- 
tions of C&I contributed to increased ‘relevance’ and ‘comparabil- 
ity’ of those C&I. Scoring against the requirements ‘practicality’ 
and ‘reliability’ was highly subjective, because the scores were 
based on the most assumptions and uncertainties in regard to data 
gathering, measuring and reliability. When scoring some C&l 
against those requirements, location-dependent levels of data 
availability and quality had to be considered, which added more 
complexity to the scoring processes. In this connection it had to be 
kept in mind, that the described scoring processes had the sole 
purpose to pre-select C&I. As described previously, participants of 
the multi-stakeholder forum had the opportunity to confirm, 
change or propose new C&I resulting in a final set. When scoring 
‘simplicity’ of C&I, the groups/organizations and positions/depart- 
ments of the participants invited to the multi-stakeholder forum 
were taken into account. However, the assumptions made when 
scoring C&I against this selection requirement can still be regarded 
as highly subjective, because knowledge and understanding of C&I 
first and foremost depend on personal backgrounds of individual 
forum participants. 


4.2. Participatory selection of environmental C&I 


In the multi-stakeholder forum, in which the final C&I took place, 
participants suggested that C&I covering biodiversity and visual 
impacts of potential plant(s) could be included. However, after a 
discussion the forum participants ruled these aspects out, because of 
the research’s scope and focus on supporting sustainability decision- 
making to screen and assess location areas, rather than specific site 
locations, for which these aspects are typically assessed. For the same 
reason, the suggestion of the regulator’s representative to look at the 
supply chain, including road transport of fuel when assessing the 


T. Kurka, D. Blackwood / Renewable and Sustainable Energy Reviews 24 (2013) 92-102 99 


Problem Hierarchy — Scenario | 


GOAL: Regional Sustainable Bioenergy 
Generation 


ENVIRONMENTAL ECONOMIC TECHNICAL SOCIAL criteria 
criteria (EN) criteria (EC) criteria (T) (S) 
EN3: EC1: EC2: T1: T2: S1: S2: 
Waste Economic Regional Efficiency echnolog Regional Regional 
Viability Energy Job Food 
Self- Creation Security 
S ficiency, 


ALTERNATIVE 1: Large-Scale 
Bioenergy Generation 


ALTERNATIVE 2: Medium-Scale 
Bioenergy Generation 


Fig. 1. Problem hierarchy for scenario 1. 


criterion ‘air quality’ was not found suitable. Thus, it was agreed that 
although assessing the three aspects is important, this would have to 
be carried out after identifying concrete plant locations. 

Contrary to that the criterion ‘GHG emissions’ and its indicator 
‘CO2 emissions’ were confirmed by the majority of the forum 
participants, although it includes assessment of CO, emissions across 
the Scottish supply chain. Forum participants from the environmen- 
tal protection organization and the bioenergy producers highlighted 
that when assessing this criterion and its indicator the bioenergy 
supply chain has to be defined by keeping the goal of the decision 
making problem, its scenarios and alternatives, as well as the ‘Life 
Cycle Assessment’ (LCA) approach and the EU Commission’s ‘Report 
on sustainability requirements for the use of solid and gaseous 
biomass sources in electricity, heating and cooling’ [53] in mind. It 
was also proposed and agreed by most experts to extent the scope of 
the indicator to ‘CO.—equivalent GHG emissions’. 

Suggestions for change and improvement were also made 
for the criterion ‘waste’ and its indicator ‘waste re-processed’. 
Although acknowledging the potential for Greenhouse Gas (GHG) 
emissions savings, the forum participant from the large scale 
bioenergy producer questioned the economic viability of re- 
processing waste materials into biofuel. On the other hand, the 
participant proposed to introduce an indicator to measure residual 
ash usage, on which the forum’s participants agreed. However, in 
terms of re-processing waste materials into biofuel, most of the 
forum participants argued that its use would be the individual 
business decision of the bioenergy producer and that the eco- 
nomic success of waste re-processor companies in Scotl- 
and demonstrated that waste re-processing can be profitable. 
In principle, it could be assumed that the reasoning of the forum 
participant representing the large-scale bioenergy producer was 
motivated to a certain extent by the organization’s plans to build a 
large-scale bioenergy plant in the research case study area in the 
future. For this plant it is planned to utilize only small amounts of 
re-processed waste as fuel and to put processes in place to make 
residual ashes useable. 


Also covering the waste aspect, a forum participant from a 
business support organization brought forward that the indicator 
and its description could be more specific in terms of waste type 
and pointed out that in re-processing waste different classifica- 
tions and regulations for various materials have to be kept in 
mind. However, it can be argued that breaking-down the several 
re-processed waste types could become relatively complex and 
less practical compared to measuring a single indicator, which 
accumulates all types of re-processed waste. This is particularly 
true when taking into account that for three of the four alter- 
natives of the case study scenarios utilization of any type of 
biomass feedstock and related technologies was possible. For this 
reason, the criterion and its indicator were not changed. 


4.3. Participatory selection of economic C&I 


The forum participants from a sawmill, a bioenergy producer and 
a business support organization raised concerns about the availability 
of biomass feedstock supplies and the increasing dependency on 
biofuel imports. They also made aware of possible consequences of 
not securing alternative, sustainable supply sources. Therefore, along 
with the substantial majority of the other experts, they agreed on the 
particular usefulness of the proposed criterion ‘regional energy 
self-sufficiency’ and its indicator ‘fuel import dependency’ covering 
this aspect. 

Moreover, in addition to the pre-selected economic C&I two 
new aspects were suggested to be incorporated during this stage 
of the multi-stakeholder forum. One aspect was about covering 
future economic incentives for the bioenergy sector and the other 
aspect about addressing the level of regulations for the different 
alternatives of the case study scenarios. However, it was found 
that both proposed aspects are directly or indirectly covered by 
the pre-selected criterion ‘economic viability’ and its indicator ‘net 
present value (NPV)’, as they either generate additional income or 
result in costs to maintain or put measures into place to comply 
with regulation. 
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Table 6 
Final set of selected C&I for the case study. 


Description 


CO2-equivalent GHG emissions across bioenergy supply chain 


Emissions of NO,, CO, SO, and particulates from operational 
bioenergy plant(s) 


% of Scottish waste re-processed to Waste re-processed to fuel for bioenergy production 


£/kW h (operational plant specific) Expected incoming and outgoing cash flows derived to present 


economic values to enable a comparison independent of time 
Percentage of imported feedstock for the bioenergy plant(s) 


Expresses the technology's ability to convert the primary energy 
source to energy (electricity and/or thermal energy) for usage 
Represents the technology’s maturity rate and its uptake/adoption 
in the international market, which reflects the technology's 
reliability 


Category Criterion Indicator Unit 
Environmental GHG emissions CO2- kg/kW h (supply chain specific) 
equivalent 
GHG 
emissions 
Air quality Non GHG- kg/kW h (operational plant 
emissions specific) 
Waste Waste re- 
processed fuel (per annum) 
Economic Economic viability Net present 
value 
Regional energy self- Fuel import % of imported feedstock 
sufficiency dependency operational plant specific) (per 
annum) 
Technical Efficiency Energy % (operational plant specific) 
efficiency 
Technology Knowledge of % (worldwide) 
technology 
Social Regional job creation Job creation Created jobs/kW h (plant and 


supply chain specific) 
% of total productive land use 
change in favour of energy crop 


Land use 
change 


Regional food security/ 
Change of landscape and 


land use plantation 


Number of direct and indirect jobs created at the bioenergy plant 
(s) and along the Scottish bioenergy supply chain 

Reflects the change of landscape and land use due to energy crop 
plantation and the availability of remaining productive land in 
Scotland 


Representatives of a bioenergy producer and a business sup- 
port organization further suggested that economic C&I covering 
risk could be considered. They argued that this would address the 
risks arising from technical failure and biofuel supply bottlenecks. 
However, it can be argued that these risks are already indirectly or 
directly addressed by the pre-selected criteria ‘regional energy 
self-sufficiency’ and ‘technology’ and their respective indicators 
‘fuel import dependency’ and ‘knowledge of technology’. There- 
fore, these suggestions for change were not further considered. 


4.4. Participatory selection of technical C&I 


The two representatives of the bioenergy producers in the multi- 
stakeholder forum outlined that the technical criterion ‘efficiency’ 
and its indicator also cover economic issues and could therefore 
categorized as either technical or economic C&I. Furthermore, they 
stated that the criterion could be renamed to ‘energy productivity’ 
to address more the possible shared use of generated heat, for 
instance for district heating, for some alternatives of the case study 
or, alternatively, the description of this criterion and indicator could 
be clearer. After the multi-stakeholder forum, it was decided that 
based on the common use and the definitions found in literature, 
the criterion and its indicator remains in the technical category and 
renaming would not be required for the case study. However, in 
order to clarify what is meant by the criterion and its indicator the 
description was amended. 

Furthermore, a forum participant representing a bioenergy 
producer further proposed that the description of the criterion 
‘technology’ and its indicator ‘knowledge of technology’ should be 
improved, as this aspect basically measures how save and reliable 
a technology is. 


4.5. Participatory selection of social C&I 


Again with respect to the supply situation, the forum participants 
from an environmental protection organization and a sawmill, sug- 
gested to take account of business and jobs taken away from large 
traditional industries in the case study area, which compete for the 
same biofuel supplies as the bioenergy industry, such as the wood 
panel industry. However, it was argued that availability of supplies 
would just be one of many factors influencing the development of this 


competitive market. Based on this argument, the majority of experts 
agreed that no practical and measurable criterion and its indicator 
would cover the issue appropriately. 

Another forum discussion followed with forum participants from 
the environmental protection organization, a local authority, a bioe- 
nergy producer and a forestry support organization suggesting that 
land use or landscape change caused by energy crop plantation could 
be addressed. As indicators ‘marginal price for farmers to switch to 
biofuels’ and ‘use of marginal land’ were proposed. However, the 
substantial majority of forum participants agreed that these indicators 
would be relatively specific and that the pre-selected criterion ‘threat 
to food security’ with its indicator and unit ‘% of total arable land used 
for energy crop plantation’ principally already covers the aspect for the 
given scenarios appropriately. By renaming the criterion and indicator, 
as well as by amending the unit and description, this was made more 
transparent and understandable. 


4.6. Final set of selected C&I 


In summary the set of pre-selected C&I (Table 5) was confirmed to 
be appropriate by a substantial majority of the participants with the 
exception of the mentioned relatively minor changes or modifica- 
tions of four C&I. Principally, the effectiveness of the pre-selection 
process was underlined by the few changes. Even in cases where no 
changes would be suggested by experts or other stakeholders in 
similar research or decision making situations, as mentioned, follow- 
ing a participatory approach is vital to ensure acceptance, transpar- 
ency and credibility throughout the decision making process. 
Furthermore, this approach encourages ownership at a local level, 
which can be crucial when it comes to the implementation phase of 
bioenergy developments, as local ownership could increase stake- 
holder acceptance leading to a higher probability of project approval 
and time savings. Although related to identified C&I such as 
‘participation’ and ‘empowerment’ (Table 1), ‘local ownership’, as 
well as ‘regional equity’ also could be considered as separate C&I for 
future research particularly focusing on decentralized and centralized 
bioenergy generation. Further notable is that qualitative C&I were not 
suggested in the multi-stakeholder forum, which is in line with the 
arguments brought forward previously (Section 3.3). Table 6 illus- 
trates the final set of selected C&I for the case study with the changes 
highlighted. In principle, it remains to be seen whether the selected 
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C&I can successfully be employed for decision making processes, if 
the described scenarios and alternatives were implemented in the 
case study area or in similar regions. This could be a subject for future 
research. 


5. Conclusions 


This paper provides a generic approach for selecting sustain- 
ability C&I for bioenergy developments by using a participatory 
methodology. Selecting appropriate C&I to assess the sustainability 
of a project or development is crucial and significantly influences 
the assessment results. 

In principle, the methodology consisted of two processes, a 
pre-selection of suitable C&I by the researchers and a final 
selection by regional bioenergy experts in a multi-stakeholder 
forum. The methodology was applied in a Scottish region (Tayside 
& Fife). 

Although the participants of the multi-stakeholder forum regarded 
the resulting set of C&I as being suitable, it remains to be seen 
whether the selected C&I can successfully be employed for decision 
making processes, if the described scenarios and alternatives 
were implemented in the case study area. However, this set of C&I 
can serve as a benchmark set in this and similar decision making 
situations. For that purpose the case study’s two scenarios and four 
alternatives were deliberately developed in a rather generic way 
allowing a wide range of application potential within the energy and 
particularly the bioenergy field. For similar research or decision 
making situations this particular set of C&I can be used as a starting 
point for C&I pre-selection prior to regional bioenergy stakeholders or 
wider stakeholders making final C&I selection decisions on a case-to- 
case basis. 

Furthermore, the literature reviews to identify relevant C&I and 
requirements for C&I selection provide a broad application poten- 
tial to assist C&I selection, as the sustainability field in general, the 
energy sector and the bioenergy sector in particular are covered. 
The same applies for the methodology itself. Although, for other 
research or decision making situations different C&I may be 
relevant, the presented approach to systematically select C&I and 
to justify this selection can be undertaken. 

The methodology for selecting C&I can be combined with 
participatory C&I prioritization to determine the relative impor- 
tance of each criterion and its indicator. This can help to improve 
performance assessments of selected C&I and support participa- 
tory decision making. 
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